Skip to content

async fixes in kernel usage handler #177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 18, 2023
Merged

async fixes in kernel usage handler #177

merged 1 commit into from
Feb 18, 2023

Conversation

minrk
Copy link
Collaborator

@minrk minrk commented Feb 16, 2023

  • handle get_msg being async or not, which I assume is sensitive to the combination of server and jupyter-client version(s)
  • use async poller to avoid blocking while waiting for usage response, removes retries

Fixes:

[E 10:21:32.039 NotebookApp] Uncaught exception GET /api/metrics/v1/kernel_usage/get_usage/a03abaf0-9f33-49f9-8bf9-515f7e64c01f?1676542891240 (172.17.0.1)
    HTTPServerRequest(protocol='http', host='127.0.0.1:58133', method='GET', uri='/api/metrics/v1/kernel_usage/get_usage/a03abaf0-9f33-49f9-8bf9-515f7e64c01f?1676542891240', version='HTTP/1.1', remote_ip='172.17.0.1')
    Traceback (most recent call last):
      File "/srv/conda/envs/notebook/lib/python3.11/site-packages/tornado/web.py", line 1713, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/srv/conda/envs/notebook/lib/python3.11/site-packages/jupyter_resource_usage/api.py", line 122, in get
        res = await client.control_channel.get_msg(timeout=0)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    TypeError: object dict can't be used in 'await' expression
[W 10:21:32.049 NotebookApp] wrote error: 'Unhandled error'
[E 10:21:32.051 NotebookApp] {
      "Host": "127.0.0.1:58133",
      "Accept": "*/*",
      "Referer": "http://127.0.0.1:58133/lab/tree/index.ipynb",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15"
    }
[E 10:21:32.051 NotebookApp] 500 GET /api/metrics/v1/kernel_usage/get_usage/a03abaf0-9f33-49f9-8bf9-515f7e64c01f?1676542891240 (172.17.0.1) 807.050000ms referer=http://127.0.0.1:58133/lab/tree/index.ipynb
[W 10:21:32.051 NotebookApp] Could not destroy zmq context for <jupyter_client.blocking.client.BlockingKernelClient object at 0xffffabe2d110>

in Python 3.11 env created with repo2docker:

Package                       Version
----------------------------- -----------
alembic                       1.9.1
anyio                         3.6.2
argon2-cffi                   21.3.0
argon2-cffi-bindings          21.2.0
asttokens                     2.2.1
async-generator               1.10
attrs                         22.2.0
Babel                         2.11.0
backcall                      0.2.0
backports.functools-lru-cache 1.6.4
beautifulsoup4                4.11.2
bleach                        6.0.0
blinker                       1.5
bokeh                         3.0.3
brotlipy                      0.7.0
certifi                       2022.12.7
certipy                       0.1.3
cffi                          1.15.1
charset-normalizer            2.1.1
click                         8.1.3
cloudpickle                   2.2.1
comm                          0.1.2
contourpy                     1.0.7
cryptography                  39.0.0
cycler                        0.11.0
cytoolz                       0.12.0
dask                          2.10.0
debugpy                       1.6.6
decorator                     5.1.1
defusedxml                    0.7.1
dill                          0.3.6
distributed                   2.10.0
entrypoints                   0.4
executing                     1.2.0
fastjsonschema                2.16.2
flit_core                     3.8.0
fonttools                     4.38.0
fsspec                        2023.1.0
greenlet                      2.0.2
HeapDict                      1.0.1
idna                          3.4
importlib-metadata            6.0.0
importlib-resources           5.10.2
ipykernel                     6.21.1
ipython                       8.9.0
ipython-genutils              0.2.0
ipywidgets                    8.0.2
jedi                          0.18.2
Jinja2                        3.1.2
json5                         0.9.5
jsonschema                    4.17.3
jupyter_client                8.0.2
jupyter_core                  5.2.0
jupyter-offlinenotebook       0.2.2
jupyter-resource-usage        0.7.0
jupyter-server                1.23.5
jupyter-telemetry             0.1.0
jupyterhub                    3.1.1
jupyterlab                    3.4.8
jupyterlab-pygments           0.2.2
jupyterlab_server             2.19.0
jupyterlab-widgets            3.0.5
kiwisolver                    1.4.4
locket                        1.0.0
Mako                          1.2.4
MarkupSafe                    2.1.2
matplotlib                    3.7.0
matplotlib-inline             0.1.6
mistune                       2.0.4
msgpack                       1.0.4
munkres                       1.1.4
nbclassic                     0.5.1
nbclient                      0.7.2
nbconvert                     7.2.9
nbformat                      5.7.3
nest-asyncio                  1.5.6
notebook                      6.4.12
notebook_shim                 0.2.2
nteract-on-jupyter            2.1.3
numpy                         1.24.2
oauthlib                      3.2.2
packaging                     23.0
pamela                        1.0.0
pandas                        1.5.3
pandocfilters                 1.5.0
parso                         0.8.3
partd                         1.3.0
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        9.4.0
pip                           23.0
pkgutil_resolve_name          1.3.10
platformdirs                  2.6.2
prometheus-client             0.16.0
prompt-toolkit                3.0.36
psutil                        5.9.4
ptyprocess                    0.7.0
pure-eval                     0.2.2
pycparser                     2.21
Pygments                      2.14.0
PyJWT                         2.6.0
pyOpenSSL                     23.0.0
pyparsing                     3.0.9
pyrsistent                    0.19.3
PySocks                       1.7.1
python-dateutil               2.8.2
python-json-logger            2.0.4
pytz                          2022.7.1
PyYAML                        6.0
pyzmq                         25.0.0
requests                      2.28.2
ruamel.yaml                   0.17.21
ruamel.yaml.clib              0.2.7
Send2Trash                    1.8.0
setuptools                    67.1.0
six                           1.16.0
sniffio                       1.3.0
sortedcontainers              2.4.0
soupsieve                     2.3.2.post1
SQLAlchemy                    2.0.1
stack-data                    0.6.2
tblib                         1.7.0
terminado                     0.17.1
tinycss2                      1.2.1
tomli                         2.0.1
toolz                         0.12.0
tornado                       6.2
traitlets                     5.9.0
typing_extensions             4.4.0
urllib3                       1.26.14
wcwidth                       0.2.6
webencodings                  0.5.1
websocket-client              1.5.0
wheel                         0.38.4
widgetsnbextension            4.0.5
xyzservices                   2022.9.0
zict                          2.2.0
zipp                          3.12.0

@minrk
async fixes in kernel usage handler
4094644 
- handle get_msg being async _or not_
- use async poller to avoid blocking while waiting for usage response
@minrk minrk added the bug label Feb 16, 2023
@welcome
Copy link

welcome bot commented Feb 16, 2023

Thanks for submitting your first pull request! You are awesome! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please make sure you followed the pull request template, as this will help us review your contribution more quickly.
welcome
You can meet the other Jovyans by joining our Discourse forum. There is also a intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

minrk
minrk commented Feb 16, 2023
View reviewed changes
jupyter_resource_usage/api.py
control_socket = control_channel.socket
poller.register(control_socket, zmq.POLLIN)
for i in range(1, MAX_RETRIES + 1):
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think a second try could ever be taken, because the options are:

  1. events is empty, in which case empty dict is written, or
  2. events contains the control socket, in which case the response is written

both of which end with break

There shouldn't be a way for the continue branch to ever be taken, which meant that the loop could only ever take one iteration.

krassowski
krassowski approved these changes Feb 18, 2023
View reviewed changes
Copy link
Collaborator

@krassowski krassowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@krassowski krassowski merged commit 7299c08 into jupyter-server:main Feb 18, 2023
@welcome
Copy link

welcome bot commented Feb 18, 2023

Congrats on your first merged pull request in this project! 🎉
congrats
Thank you for contributing, we are very proud of you! ❤️

@krassowski krassowski mentioned this pull request Feb 18, 2023
Open
@viniciusdc viniciusdc mentioned this pull request Mar 28, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@krassowski krassowski krassowski approved these changes

Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@minrk @krassowski